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ABSTRACT 


’’This  paper  reports  the  status  of  Phase  I of  an  ongoing  project  to  develop  a 
macro  model  describing  the  decisions  involved  in  developing  training  equipment. 
The  purpose  of  the  model  is  to  assist  managers  in  making 'such  decisions  by  providing 
information  concerning  the  tradeoffs  between  cost  and  training  effectiveness  caused 
by  different  configurations  and  choices  of  equipment.  After  the  development  of  a 
preliminary  model,  field  research  was  conducted  to  determine  the  feasibility  of  test- 
ing such  a model  and  to  collect  information  to  expand  the  preliminary  version  into  a 


more  pragmatic  tool 


Results  of  the  field  work  led  to  several  conclusions.  First,  many  of  the  types 
types  of  data  needed  to  validate  such  a model  are  available,  hence  making  such 
a project  feasible.  Second,  an  examination  of  the  available  data  led  to  an  expansion 
of  the  preliminary  model  to  include  training  value  ofthe  various  trainer  characteristics. 
Third,  much  work  is  needed  to  develop  longitudinal  data  bases  of  job  performance 
before  sound  predictions  can  be  made  concerning  the  impact  of  trainer  characteristics 
on  technician  performance  after  graduation. 


PURPOSE 


BACKGROUND 


This  paper  reports  the  status  of  an  ongoing 
project  to  develop  a macro  model  to  assist 
managers  in  making  decisions  concerning  train- 
ing equipment.  The  model  is  designed  to  permit 
comparisons  of  alternative  equipment  and  con- 
figurations, and  the  associated  costand  training 
effectiveness  for  the  alternatives.  The  project 
was  designed  to  meet  the  three  objectives  listed 
below: 


(1) 


To  develop  a macro  model  for  use  in 
making  cost/benefit  tradeoffs  between 
the  various  characteristics  that  may 
be  utilized  in  training  equipment,  and 
to  give  both  military  and  industry 
guidelines  to  justify  decisions  relating 
to  trainer  design. 


The  operational  readiness  of  sophisticat- 
ed military  systems  depends,  to  a great  extent, 
on  the  skills  of  the  technicians  who  maintain 
those  systems.  As  the  sophistication  of  current 
military  equipment  increases,  the  operational 
training  for  the  use  of  such  equipment  becomes 
simpler.  Conversely,  the  training  of  technicians 
to  maintain  the  equipment  becomes  an  even  more 
complex  task  (1).  In  order  to  assist  in  the 
training  of  maintenance  personnel,  therefore, 
the  Armed  Services  are  increasingly  relying  on 
the  use  of  maintenance  trainers  and  simulators 
(2). 


(2)  To  determine  the  efficiency  of  a sam- 
ple of  trainers  currently  in  use. 


<3) 


To  determine  the  relative  cost  and 
training  effectiveness  of  various 
characteristics  or  capabilities  that  can 
be  used  in  training  equipment. 


* The  authors  would  like  to  express  their 
appreciation  to  Janet  L.  Blanche  for  her 
assistance  during  the  preparation  of  this 
manuscript. 


There  are  two  basic  reasons  underlying 
this  trend.  First  of  all,  it  is  assumed  that 
training  devices  are  capable  of  training  a stu- 
dent at  least  as  effectively  as  instruction  which 
only  utilizes  actual  equipment.  In  fact,  research 
in  instructional  psychology  suggests  that  the  use 
of  actual  equipment  for  training  purposes  may 
be  less  efficient  and  effective  than  the  use  of 
training  devices  which  permit  greater  latitude 
in  material  presentation  (3).  For  example,  a 
training  device  can  be  designed  so  that  an  over- 
all task  is  broken  into  smaller,  less  complicated 
subtasks  which  can  be  more  readily  grasped  by 
a student.  In  theory,  this  leads  to  a better  even- 
tual mastery  of  the  whole  task  (4).  Training 
devices  also  allow  students  to  practice  and  ob- 
serve procedures  which,  although  necessary  for 
competency,  are  potentially  too  dangerous  in 
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terms  of  personnel  and/or  equipment  safety  for 
the  neophyte  to  perform  in  the  training  environ- 
ment (5).  Finally,  training  devices  are  assumed 
to  provide  more  efficient  training  due  to  their 
built-in  capabilities  which  allow  automatic  stu- 
dent monitoring,  decreased  instructor  demand 
for  routine  problems,  and  increased  student 
experience  and  practice  (6). 

The  second  reason  why  training  devices 
are  becoming  increasingly  popular  is  that  such 
devices  are  often  assumed  to  be  more  costeff"''- 
effective  than  the  use  of  actual  equipment  (’). 
The  use  of  actual  equipment  in  training  situa- 
tions is  expensive  in  terms  of  such  factors  as 
fuel  costs,  equipment  wear  and  tear  or  damage, 
and  loss  of  availability  for  operation  whilediver- 
ted  to  training  use.  Therefore,  even  when  the 
training  effectiveness  of  a course  utilizing  a 
training  device  is  only  equivalent  to  that  of  a 
course  utilizing  the  actual  equipment,  if  the 
device  is  less  expensive  to  acquire  and  run 
than  the  actual  equipment,  it  is  a better  value 
for  training  purposes, 

Although  such  reasons  for  using  training 
devices  are  seductive  in  theory,  these  assump- 
tions have  not  necessarily  been  found  to  be  true 
in  practice.  Therefore,  the  question  that  arises 
is  whether  or  not  training  devices  are  effec- 
tive --either  in  terms  of  cost  or  training  -- 
when  compared  with  more  traditional  training 
methods.  The  principles  of  learning  psychology 
have  been  found  to  fall  short  when  practiced  in 
military  training  situations  (8).  Many  of  the 
"common  sense"  characteristics  often  included 
in  the  design  of  training  devices  ,e.g.  , high 
realism)  have  not  been  found  to  be  consistently 
effective  (9,10).  Furthermore,  although  the 
primary  goal  of  training  is  to  produce  qualified 
technicians  more  quickly,  only  one  study  has 
been  performed  to  comparatively  evaluate  the 
effectiveness  of  technicians  trained  with  actual 
equipment  versus  those  trained  using  simulators . 
No  differences  were  found  between  groups  in 
this  study  (11).  Virtually  no  research  has  been 
conducted  relating  life  cycle  costs  to  training 
effectiveness. 

The  lack  of  hard  facts  concerning  the  use- 
fulness of  various  trainer  characteristics  makes 
training  device  justification  and  design  tradeoffs 
difficult.  Because  of  this,  decisions  are  made 
based  only  on  cost  factors  since  there  are  no 
data  available  to  evaluate  training  value.  There- 
fore, both  the  Services'  project  managers  and 
industry  have  been  unable  to  articulate  require- 
ments for  training  devices  in  terms  of  projected 
training  value.  Equipment  for  training  is  often 
proposed,  developed,  and  procured  without  an 
estimate  of  the  expected  training  effectiveness 
ana  wunoui  a tnorougn  ana*ysis  of  wnicn  tasxs 
are  suited  to  the  use  of  a training  device  or  sim- 
ulator. Such  practices,  however,  mean  that  the 
creation  of  an  optimally  effective  training  device 
is  only  serendipitous. 

Since  effective  training  is  vital  for  the  devel- 
opment of  the  skilled  technicians  who  are  a key 
link  in  the  chain  of  operational  readiness  (12), 
it  is  necessary  that  incompatibilities  between 
current  assumptions  and  actual  data  be  identified 
and  rectified.  There  is,  therefore,  a strong 


requirement  that  training  efficiency  and  effec- 
tiveness be  considered  along  with  costs  during 
the  front  end  analysis  and  design  of  new  mainte- 
nance training  equipment.  The  formulation  of 
consistent  and  effective  policies  for  such  designs 
depends  to  a great  extent  on  the  use  of  cost  and 
training  effectiveness  data  in  a model  of  logis- 
tics, manpower,  personnel,  and  training.  The 
projections  of  such  a model  must  be  further 
clarified  and  validated  using  hard  data  collected 
from  the  field. 

Two  sets  of  factors  must  be  dealt  with  in 
such  a project:  economic  considerations  and 

learning  quality.  Historically,  cost  analysis  has 
been  accomplished  for  tradeoffs  between  actual 
equij  ment  and  simulators  used  for  training.  Cost 
cons  derations  have  been  based  on  investment 
costs  and  efficiency  items  such  as  expected  fuel 
savings.  However,  experience  with  successful 
training  devices  indicates  that  certain  effici- 
encies which  can  also  be  assigned  a dollar 
value  accrue  once  the  trainer  is  fielded.  Exam- 
ples of  these  are  time  saved  in  training,  opera- 
tional equipment  made  available  for  missions 
other  than  training,  lowered  investment  costs , 
reduced  maintenance  costs,  and  less  run  time 
on  operational  equipment.  Such  potential  effi- 
ciencies should  be  identified  and  developed  into 
an  economic  model  for  use  in  prediction  of  life 
cycle  costs  during  front  end  analysis. 

The  second  set  of  factors  which  must  be 
considered  in  such  a project  deals  with  differ- 
ences in  training  effectiveness  between  courses 
using  training  devices  and  those  using  the  actual 
equipment.  Often  the  assumption  is  incorrectly 
made  that  if  a training  device  is  less  expensive 
than  the  actual  equipment  as  used  in  training, 
it  is  a better  buy,  and,  therefore,  should  be 
used.  This  line  of  reasoning  is  specious.  In 
order  to  justify  the  use  of  a training  device, 
it  is  fallacious  to  consider  only  the  costs  of 
acquiring  and  maintaining  it.  Attention  must 
also  be  focused  on  how  well  each  facilitates 
student  learning,  and  on  the  resulting  benefit 
of  more  quickly  producing  qualified  technicians. 
While  effectiveness  is  not  easily  quantifiable, 
there  are  enough  indicators  to  believe  that,  at 
least  in  gross  terms,  it  can  be  measured  and 
modeled.  Such  information  would  be  helpful  in 
making  sound  management  decisions  on  the  value 
of  simulators  for  training. 


The  evaluation  of  the  cost  and  training  effec- 
tiveness of  rratntenanct  simulators  and  trainers 
is  not  a new  idea.  In  past  studies,  investigators 
have  typically  examined  variables  such  as  ac- 
quisition costs  and  training  effectiveness  in 
the  scnuul  setting  lid;.  Mmiary  training.  How- 
ever, is  intended  to  prepare  personnel  to  per- 
form various  jobs  in  operational  units.  There- 
fore, the  true  measure  of  effectiveness  is  actual 
job  performance  rather  than  performance  with 
in  a training  course.  Similarly,  although  com- 
parisons between  acquisition  costs  of  training 
devices  versus  other  training  methods  are  an 
important  consideration,  it  is  also  necessary 
to  consider  the  comparative  life  cycle  costs  of 
these  methods  before  an  informed  decision  can 
be  made.  Such  an  examination  must  be  accom- 
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plished  over  a broad  spectrum  of  applications 
before  an  ultimate  criterion  of  training  program 
effectiveness  can  be  established  or  predicted. 

Two  types  of  tools  are  necessary  in  order 
to  make  sound,  knowledgeable  decisions  during 
the  development  of  maintenance  trainers.  The 
first  of  these  tools  is  a macro  model  which  can 
be  used  as  a high  level  filter  in  making  design 
decisions  and  determining  device  justifiability. 
This  type  of  model  would  act  as  a decision 
screen,  specific  only  enough  for  early  decision 
making  by  program  managers.  Concomitant  with 
such  a decision  screen,  a more  specific,  prag- 
matic design  screen  must  also  be  developed. 
This  type  of  tool  could  be  used  effectively  both 
on  a lower  level  in  Do  D and  in  industry  for  making 
specific  cost/training  effectiveness  tradeoffs  in 
the  design  of  future  training  devices. 

PHASE  I:  FIELD  STUDY 

Phase  I of  this  project  was  designed  as  a 
pilot  study  to  identify  potential  efficiency  and 
effectiveness  factors,  and  to  develop  a proto- 
type model  with  them.  Maintenance  trainers 
were  chosen  for  analysis  in  this  phase  of  the 
study  for  the  following  reasons: 

(1)  Cost  indicators  and  potential  savings 
are  not  obvious  as  in  the  case  of  flight 
simulators  where  flying  hours  have  a 
known  cost.  Thus,  the  study  avoids  the 
issues  of  expressing  cost  avoidance  as 
an  efficiency,  and  permits  a focus  on 
more  subtle  cost  savings  as  well  as 
highlighting  concern  on  effectiveness 
issues . 

(2)  Instances  exist  in  maintenance  train- 
ing where  individuals  in  the  same  skill 
and  weapon  system  were  trained  indif- 
ferent ways  --  some  with  actual  equip- 
ment and  others  with  training  devices. 
This  situation  allows  the  collection  of 
control  data  better  suited  for  making 
judgments  of  the  relative  cost  and  train- 
iTig  effectiveness  of  cuui  sta  using  t.  din- 
ing devices  versus  those  which  do  not. 

(3)  The  factors  leading  to  the  design  of  an 
effective  maintenance  trainer  are  much 
Less  Well  defined  Than  toi  operatui 
trainers.  Therefore,  if  the  character- 
istics leading  to  effective  design  can 
be  determined  and  put  into  a model  for 
maintenance  trainers,  it  is  probable 
that  attempts  to  generalize  such  a 
model  would  be  much  less  difficult  than 

fho  mnHol  u/o np  fi  rat  H pvp  1 np^H  fr>r 

operator  trainers. 

(4)  Emphasis  on  ways  to  enhance  mainte- 

nance training  to  offset  design,  support, 
and  manpower  problems  has  forced  DoD 
to  look  at  ways  to  improve  and  expedite 
support  training.  Selection  of  mainte- 
nance trainers  for  this  work  is  consis- 
• « f"  T\  n d»’l  ■ he  Hi*  k.  ' re  dove 

training  for  maintenance. 


Preliminary  Model 

As  discussed  earlier  in  this  paper,  in  or- 
der to  develop  a valid  tool  to  make  knowledge- 
able decisions  concerning  device  cost  and 
training  effectiveness  tradeoffs,  four  levelB  of 
variables  must  be  considered: 

(1)  Training  effectiveness  in  the  school: 

(2)  Training  effectiveness  in  the  field; 

(3)  Acquisition  costs  of  equipment; 

(4)  Life  cycle  costs  of  equipment. 

However,  although  necessary  in  the  creation 
of  such  a tool,  these  factors  are  not  sufficient 
in  and  of  themselves.  There  are  also  several 
modifying  variables  which,  although  not  directly 
relating  to  device  effectiveness,  act  as  filters 
integral  to  any  comparison  between  the  effec- 
tiveness of  a training  device  versus  the  actual 
equipment  (Figure  1).  First  of  all,  the  goals  of 
various  maintenance  training  programs  differ, 
and  the  principles  of  instructional  technology 
do  not  lend  support  to  the  supposition  that  the 
same  type  of  training  will  be  equally  efficient 
for  all  purposes.  (For  example,  if  the  goal  is 
to  teach  the  student  motor  skills  suen  as  per- 
forming a task  requiring  interactive  analog  in- 
puts, one  would  probably  not  depend  wholly  on 
a verbal  explanation  --  without  hands-on  prac- 
tice --  to  teach  such  a task.)  Therefore,  in 
order  to  compare  the  relative  effectiveness  of 
training  programs,  it  is  important  to  keep  in 
mind  the  objectives  toward  which  the  learning 
situations  are  aimed.  A second  filter  which 
must  be  considered  in  the  creation  of  such  a 
model  is  the  characteristics  of  the  students 
being  taught.  For  example,  in  teaching  a low 
level  student  with  vitually  no  experience  or 
knowledge  in  the  subject  area  it  would  be  nec- 
essary to  go  into  much  more  detailed  explana- 
tions of  the  subject  matter  than  when  giving 
a review  course  to  students  who  had  been  per- 
forming the  same  task  for  over  a year.  A third 
filter  which  one  must  be  aware  of  in  facing 
such  comparisons  is  the  characteristics  of  the 
instructor  teaching  the  course.  For  example, 
if  ail  instructor  dues  not  understand  or  like  to 
use  a particular  technique  for  teaching,  he  or 
she  will  not  utilize  the  method  as  well  as  an 
instructor  who  does. 

iherelore,  although  comparing  the  rela- 
tive effectiveness  of  maintenance  trainer  equip- 
ment and  actual  equipment  training  is  theoreti- 
cally a straightforward  task,  it  is  important 
to  consider  these  modifying  variables  --program 
goals,  student  experience  and  aptitude,  and  the 
instructor  experience  and  aptitude  --in  order 
.^oj-ii.-gfu'iy  'nterp'vt  any  results  of  such 
a comparison  ( Figure  1). 

This  framework  was  used  as  a prelimin- 
ary paradigm  to  describe  the  field  of  training 
device  cost  and  training  effectiveness  factors. 
The  Phase  I field  study  was  then  designed  to 
test  the  effect  of  the  independent  variable  of 
training  method  on  the  four  levels  of  device 
«AicUiMitie»6.  Iskuyg  unlto  tjccuM.  Cto  tT&Ubf  - 
ing  variables  discussed  above.  The  objectives 
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of  this  field  study  were  to  collect  information 
to  develop  this  initial  paradigm  into  a testable 
model,  and  to  determine  the  feasibility  of  collec- 
ting data  to  test  the  model. 

Methodology 

To  help  achieve  these  objectives,  two  sets 
of  data  collection  instruments  were  developed. 
The  first  of  these  was  a set  of  Behaviorally 
Anchored  Rating  Scales  (BARS)  which  were  used 
to  assess  technicians'  performance  in  the  the 
field.  In  order  to  develop  these  scales,  instruc- 
tors in  the  role  of  subject  matter  experts  were 
asked  to  create  a series  of  critical  incidents 
describing  behaviors  which  differentiate  between 
a good  technician  and  a poor  one.  These  inci- 
dents focused  on  specific  technician  actions 
closely  related  to  the  job,  and  differentiated 
between  success  and  failure  as  a maintenance 
technician.  Several  hundred  of  these  incidents 
were  collected,  thematically  analyzed,  and 
placed  into  scales.  They  were  then  rated  by 
the  instructors  on  a seven -point  scale  with  the 
scale  value  of  1 being  very  poor  performance 
behavior  and  the  scale  value  of  7 being  very 
high  performance  behavior.  Those  incidents 
with  the  lowest  standard  deviations  and  means 
closest  to  1 and  to  7 were  then  placed  on  a 
graphic  type  rating  scale  to  be  used  as  behav- 
ioral anchors  for  the  scale. 

There  are  two  advantages  to  using  BARS. 
First,  the  description  of  the  scale  points  is  writ- 
ten in  terms  that  can  be  easily  understood  by 
the  raters.  Second,  since  the  type  of  person 
who  developed  the  scale  is  also  the  type  of  per- 
son who  uses  the  scale,  the  raters  have  a vested 
interest  in  using  the  scales  correctly  (15). 

The  use  of  the  BARS  development  techni- 
que in  this  study  yielded  seven  specific  scales: 


Safety:  Behaviors  which  show  that  the 

technician  understands  and  follows 
safety  practices  as  specified  in  the 
technical  data: 


Thoroughness  and  Attention  to  Details: 
Behaviors  which  show  that  the  techni- 
cian is  well  prepared  when  he  arrives 
on  the  job,  carries  out  maintenance 
procedures  completely  and  thoroughly, 
and  recognizes  and  attends  to  symptoms 
of  equipment  damage  or  stress; 

UBe  of  Technical  Data:  Behaviors 

which  show  that  the  technician  proper- 
ly uses  technical  data  in  performance 
of  maintenance  functions; 

System  Understanding:  Behaviors 
which  show  that  the  technician  thor- 
oughly understands  system  operation 
allowing  him  to  recognize,  diagnose, 
and  correct  problems  not  specifically 
covered  in  the  Technical  Orders  and 
publications; 

Understanding  of  Other  Systems:  Be- 
haviors which  show  that  the  technician 
understands  the  systems  that  are  inter- 
connected with  his  specific  system  and 
can  operate  them  in  accordance  with 
technical  orders; 

Mechanical  Skills:  Behaviors  which 

show  that  tKe  Technician  possesses 
specific  mechanical  skills  acquired 
for  even  the  most  difficult  mainten- 
ance problems;  and 

Attitude:  Behaviors  which  show  that 

the  technician  is  concerned  about  prop- 
erly completing  each  task  efficiently 
and  on  time. 


These  seven  skills  were  found  to  be  generic, 
holding  true  for  both  systems  utlimately  chosen 
for  this  study  phase. 

The  second  data  collection  instrument  was 
a series  of  questionnaires  for  students,  Instruc- 
tors, and  technicians.  These  questionnaires 
were  designed  to  collect  two  types  of  informa- 
tion: (1)  demographic  information  on  subject 
background,  training,  and  experience:  and  (2) 
subjective  information  such  as  subjects'  attitudes 
toward  training  devices  in  general,  and  their 
perceptions  and  evaluations  of  the  specific  device 
with  which  they  were  working. 

For  this  phase  of  the  study,  the  F-16  and 
E-3A  AWACS  Navigation  Maintenance  trainers 
were  selected  to  serve  as  the  data  sources  for 
the  development  of  the  preliminary  model.  These 
lr„hw«*  Sw,  of  the  iivMf  exhtaptes 

of  maintenance  trainers  in  the  field.  The  F-16 
trainer  consists  of  a series  of  six  freeplay 
systems  designed  to  assist  in  teaching  mainte- 
nance courses  in  the  flight  controls.  Comm/ 
Nav,  electrical  systems,  engine  start,  engine 
diagnostics,  and  engine  run  for  the  F-16  air- 
craft. The  AWACS  device  is  a procedural  train- 
er designed  to  be  used  in  courses  in  navigation- 
al system  maintenance. 

Data  for  this  phase  of  the  study  were  collec- 
ted from  subjects  who  had  used  these  two  training 
devices,  and  also  from  control  subjects  who  had 
gone  through  the  same  courses  without  using 


the  training  devices.  The  training  effectiveness 
soft  data  were  collected  using  the  three  versions 
of  the  questionnaire  to  gather  background  data 
concerning  the  subjects  and  their  opinions  of 
training  courses  and  devices.  The  BARS  were 
used  to  determine  tne  instructors'  performance 
appraisals  of  those  students  having  just  com- 
pleted the  course  and  of  those  having  previously 
taken  the  courses.  Supervisors'  performance 
appraisals  were  obtained  for  technicians  having 
recently  graduated  the  courses,  and  for  those 
who  had  previously  graduated  the  courses.  This 
redundant  uBe  of  the  BARS  was  done  in  order 
to  help  determine  the  validity  of  such  subjective 
judgments,  and  to  partially  ascertain  the  rela- 
tionship between  judgments  of  technician  per- 
formance at  the  school  and  field  levels. 
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Hard  data  of  training  effectiveness  were 
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and  Work  Unit  Code  (WUC)  information.  WUCs 
yield  hard  data  concerning  group  and  individual 
level  technician  performance  ia  the  £u?i.d  in  eurh 
areas  as  removal  time,  repair  time,  retest  OK 
(RETOK)  rates,  etc. 


Data  concerning  the  acquisition  and  life 
cycle  costs  for  courses  utilizing  and  not  utiliz- 
ing the  trainer  were  also  collected  through  the 
Air  Force  and  the  contractor.  These  field  study 
data  are  summarized  in  TableB  1 and  2.  Miss- 
ing data  in  the  tables  result  from  the  normal 
vicissitudes  of  subject  availability  encountered 
in  a field  study  of  this  nature. 


Table  1.  Data  Collected  In  Phase  I Field  Study 


TRAINING  EFFECTIVENESS  STUDY 

School 

Field 

F-16 

AWACS 

F-16 

AWACS 

Soft  Data 

Soft  Data 

Soft  Data 

Soft  Data 

• Instructor  Inter- 

e Instructor  Inter- 

t Technician  Inter- 

• Technician  Inter- 

views 

views 

views 

views 

■ Student  Interviews 

• 

i Instructor  Question- 

• Instructor  Question- 

• Technician  Question- 

• Technician  Question- 

naires 

naires 

naires 

naires 

• Student  Questionnaires 

• 

l Instructor's  Perfor- 

• Instructor's  Perfor- 

• Supervisor's  Perfor- 

• Supervisor's  Perfor- 

mance  Appraisal  of 

mance  Appraisal  of 

mance  Appraisal  of 

mance  Appraisal  of 

Student  (BARS) 

Student  (BARS) 

Technician  (BARS) 

Technician  (BARS) 

Hard  Data 

Hard  Data 

Hard  Data 

Hard  Data 

i Test  Scores 

• Test  Scores 

• Work  Unit  Code  (WUC's) 

• Work  Unit  Code  (WUC's) 

Information 

Information 
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Table  2.  Data  Collected  In  Phase  I Field  Study 


COST  DATA 


Acquisition 


Life  Cycle 


Hard  Data 


Hard  Data 


Hard  Data 


Hard  Data 


• Developmental  Costs  • Developmental  Costs  • Consummate  Materials  • Consummate  Materials 


t Equipment  Costs 

• Instructor  Training 
Costs 


• Equipment  Costs 

• Instructor  Training 
Costs 


• Rate  of  Revision 


• Rate  of  Revision 


• Equipment  Maintenance  • Equipment  Maintenance 


• Instructor  Preparation  • Instructor  Preparation  • Equipment  Avail- 

Time  Time  ability 

• Area  Requirements  • Area  Requirements  • Life  Span  of 

Equipment 

• Fuel /Support 
Equipment  Costs 


• Area  Requirements 


• Equipment  Avail- 
ability 

• Life  Span  of 
Equipment 

• Fuel /Support 
Equipment  Costs 


• Instructor  Prepara-  • Instructor  Prepara- 


tion Time 


tion  Time 


Revised  Model 

Although  it  was  recognized  during  the  con- 
struction of  the  preliminary  model  that  trainer 
characteristics  are  an  important  variable  direc- 
tly impacting  the  effectiveness  of  training  de- 
vices, a comprehensive  list  of  these  character- 
istics could  not  be  developed  until  the  views  of 
the  users  were  collected  and  the  impact  of  the 
various  charateristics  on  effectiveness  were 
observed.  This  information  showed  that  by  and 
large  the  greatest  "need"  perceived  by  users 
was  for  high  "realism".  As  stated  earlier  in 
this  paper,  training  literature  does  not  totally 
support  such  a statement  of  need.  However, 
after  close  examination  of  the  data,  it  was  found 
that  the  discrepancies  between  the  findings  in 
the  literature  and  the  "universal"  statements 
by  the  users  were  to  a great  extent  a result  of 
the  lack  of  a consistent  definition  for  the  term 
"realism".  For  example,  when  one  instructor 
stated  the  need  for  high  "realism"  in  the  design 
of  an  engine  trim  box  to  teach  a motor  skill,  he 
was  not  necessarily  referring  to  the  same  de- 
vice characteristic  as  an  instructor  asking  for 
high  "realism"  of  engine  sounds.  As  a result, 
the  characteristic  of  "realism"  was  replaced 
for  the  purposes  of  this  model  by  the  taxonomy 
listed  below. 

1 . Static  Realism 

a.  Visual  Realism:  The  extent  to 

which  the  device  components  or 
subsystems  appear  to  be  the  same 
as  on  the  actual  equipment. 


Spatial  Realism: 


The  extent  to 


which  the  device  components  or 
subsystems  are  physically  situ- 
ated as  on  the  actual  equipment. 

c.  Auditory  Realism:  The  extent  to 

which  the  device  components  or 
subsystems  approximate  the  sounds 
of  the  actual  equipment. 

d.  Kinesthetic  Realism:  The  extent 

to  which  the  device  components  or 
subsystems  approximate  the  feel 
of  the  actual  equipment. 

2 . Dynamic  Realism 

a.  Temporal  Realism:  The  extent 

to  which  the  reaction  time  and  re- 
sponse of  the  device  components, 
or  subsystems  approximate  the 
actual  equipment. 

b.  Extent  of  Simulation:  The  extent 

to  which  the  device  as  a system 
approximates  the  total  responses 
of  the  actual  equipment  rather  than 
only  following  the  responses  given 
in  the  technical  data. 

The  other  training  device  characteristic 
which  many  users  felt  was  important  to  consider 
was  computer  aided  instruction  (CAI).  This  too 
appeared  to  be  a taxonomy  rather  than  a single 
variable: 

1 . Student  Aids:  Those  computer  man- 

aged functions  which  directly  aid  the 
student  to  learn  the  material. 
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2.  Instructor  Aids:  Those  computer  man- 

aged functions  which  indirectly  aid  the 
instructor  in  teaching  the  student  the 
material. 

3.  User  Aids:  Those  computer  manag- 

ed functions  which  facilitate  the  stu- 
dent's use  of  the  training  device. 

Based  on  these  taxonomies  and  the  infor- 
mation collected  in  the  field,  the  preliminary 
model  was  revised  to  expand  the  description 
of  the  parameters  to  be  considered  in  the  design 
and  development  of  a training  device.  (This  re- 
vised model  is  graphically  depicted  in  Figure 
2.)  The  subjective  data  collected  supported 
the  hypothesis  of  the  influence  of  modifying  var- 
iables on  training  effectiveness.  Of  the  three 
original  modifying  variables  considered,  sup- 
port was  found  to  justify  further  investigation 
of  the  effects  of  both  student  characteristics 
and  training  program  goals  on  training  effec- 
tiveness. Information  about  these  two  charac- 
teristics consistently  suggested  that  "lower 
level"  students  --where  this  is  defined  by  skill, 
knowledge,  or  experience  -can  best  be  taught 
using  different  techniques  than  those  used  for 
"higher  level"  students.  Similarly,  the  data 
suggested  that  different  sets  of  device  charac- 
teristics would  best  be  utilized  in  teaching  dif- 
ferent program  goals  (e.g.  , mechanical  skills 
or  theoretical  troubleshooting) . 

The  attitudes  of  an  instructor  toward  a 
training  device,  however,  (although  logically 
related  to  how  well  the  device  is  used  by  the 
instructor  and  how  well  the  student  uses  and 
learns  from  the  device)  were  not  found  to  sig- 
nifically  impact  training  effectiveness  in  the 
sites  and  programs  investigated  in  Phase  I. 
In  the  two  programs  studied,  the  instructors 
as  a whole  were  rather  ambivalant  towards  the 
devices  when  they  first  began  to  use  them,  but 
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with  the  devices  increased.  Although  this  may 
cause  differential  effects  in  training  between 
students  taught  directly  after  trainer  acquisi- 
tion as  opposed  to  those  taught  later,  the  effects 
of  this  variable  appear  to  be  equated  over  a per- 
iod time.  A lesson  to  be  learned  here,  however, 
is  that  improved  instructor  training  immediate- 
ly following  the  fielding  of  the  training  device 
can  positively  impact  the  acceptance  and  use 
of  the  device. 

Although  the  modifying  variables  mention- 
ed above  have  significant  impact  on  the  quality 
of  training  and  must  be  considered  in  the  inter- 
pretation of  effectiveness  data , the  characteris- 
tics of  the  device  itself  are  of  superordinate 
importance  in  their  effect  on  cost  and  training 
effectiveness.  For  the  purpose  of  this  model, 
"effectiveness"  is  defined  as  a relative  term. 
Since  training  effectiveness  cannot  currently 
be  judged  on  an  absolute  scale,  it  must  be 
examined  through  a comparison  between  the 
quality  of  technicians  in  the  field  who  have  been 
taught  using  a training  device  versus  those  who 
have  not.  The  relationship  of  these  character- 
istics to  the  overall  model  is  represented  to 
the  horizontal  "slice"  depicted  in  Figure  2.  In 
Figure  3,  one  of  these  "slices"  has  been  rotated 
90  degrees  in  order  to  give  a clearer  view  of  its 
components. 


At  this  level,  the  model  can  potentially  aid 
in  making  more  pragmatic  decisions  concern- 
ing the  design  of  training  devices.  Given  a Bound 
definition  of  training  device  cost  (as  defined 
both  by  acquisition  and  life  cycle  costs),  and 
the  establishment  of  a sufficient  data  base,  it 
will  be  relatively  easy  to  determine  the  cost 
associated  with  different  training  device  de- 
signs under  consideration.  For  example,  in 
low  quantities,  those  characteristics  defining 
static  realism  are  relatively  low  in  cost  as 
opposed  to  those  characteristics  defining  dynam- 
ic realism.  The  missing  factor  in  the  design 
and  development  considerations  of  today,  how- 
ever, is  the  concomitant  consideration  of  com- 
parative training  effectiveness.  By  developing 
a large  enough  data  base,  it  will  eventually 
be  possible  to  generate  a series  of  equations 
which  will  allow  managers  to  make  decisions 
as  to  the  relative  training  effectiveness  of  var- 
ious alternative  design  configurations  under 
consideration.  It  will  then  be  possible  to  deter- 
mine not  only  the  cost  of  acquiring  and  using  a 
training  device,  but  also  the  training  effective- 
ness of  such  a device  relative  to  the  actual 
equipment.  For  example,  in  Figure  3,  for  a 
certain  training  goal  and  student  level  there 
are  hypothetically  two  different  configurations 
which  could  be  used  in  the  design  and  develop- 
ment of  a trainer,  both  of  which  would  be 
equivalent  to  each  other  and  to  the  actual  equip- 
ment in  terms  of  training  value.  The  recom- 
mendation would  then  be  to  either  use  the 
more  cost  effective  of  the  two  alternatives, 
or  to  increase  the  projected  level  of  training 
effectiveness  by  selecting  a device  configura- 
tion higher  on  the  Y-axis. 

The  development  of  this  paradigm  into  a 
useable  tool  for  managers  depends  upon  the 
creation  of  performance  measures  to  determine 
the  training  value  of  various  design  character- 
istics. Once  this  is  accomplished,  tradeoffs 
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determined.  It  should  be  recognized  that  the 
design  characteristics  for  a training  device 
or  simulator  are  the  driving  factors  in  effec- 
tiveness potential  and  are  also  the  major  deter- 
minants of  the  utlimate  cost.  This  leads  to  the 
conclusion  that  cost  and  benefit  tradeoffs  can 
be  made  for  design  characteristics  once  the 
manager  determines  specific  training  objec- 
tives. It  will  then  be  possible  to  approach 
the  larger  question  of  comparing  cost  and 
training  effectiveness  in  order  to  make  a ration- 
al choice  between  equipment  alternatives. 

PHASES  II  AND  III:  MODEL 
REFINEMENT  AND  VALIDATION 

The  model  as  described  above  was  develop- 
ed using  two  maintenance  trainers.  Before  it 
can  be  considered  as  a viable  instrument  which 
can  be  applied  to  other  maintenance  training  de- 
vices it  must  undergo  validation  and  refinement. 
To  do  this,  it  must  be  ascertained  whether  the 
parameters  of  the  model  validly  apply  to  other 
maintenance  training  device  systems,  whether 
the  indicators  of  cost  and  training  effectiveness 
are  meaningful  for  the  evaluation  of  mainte- 
nance training  devices,  and  whether  the  design 
characteristics  included  in  the  model  signifi- 
cantly impact  the  usefulness  of  the  device  in 
terms  of  these  indicators. 
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Although  this  model  has  intuitive  appeal, 
it  cannot  be  used  until  it  is  validated.  In  the 
future,  it  will  be  necessary  to  collect  longitud- 
inal data  along  various  points  of  the  character- 
istics continua  and  relate  these  to  both  cost 
and  training  effectiveness.  Such  a validation 
must  be  longitudinal,  to  say  the  least,  and  is 
outside  the  scope  of  the  present  study.  Valida- 
tion at  this  point  in  time  will  consist  of  collect- 
ing comprehensive  data  on  the  training  systems 
currently  in  the  field.  The  systems  will  be  rated 
rated  and  placed  on  the  characteristics  continua, 
and  used  as  inputs  for  the  model  validation  and 
refinement. 

The  next  step  will  be  to  collect  data  from 
courses  using  maintenance  training  devices  and 
their  concomitant  control  courses . These  data 
will  be  analyzed  in  terms  of  the  structure  of 
the  model  to  test  for  goodness  of  fit.  The  model 
will  then  be  refined  in  order  to  better  account 
for  all  data  based  on  any  mismatches  in  the 
comparison.  Once  the  model  has  been  refined 
in  Phase  11,  it  will  be  further  validated  using 
a broader  spectrum  of  training  devices  and 
simulators.  Much  is  already  known  about  re- 
quirements for  development  and  design  of 
an  effective  operator  trainer.  Phases  I and  II 
of  the  current  study  will  offer  significant  parallel 
data  for  maintenance  trainers.  This  information 
will  be  synthesized  and  validated  for  the  crea- 
tion of  a generic  model  in  Phase  III. 

Several  types  of  questions  are  to  be  answer- 
ed in  Phase  III.  If  the  indicators  of  cost  and 
training  effectiveness  are  found  to  be  valid  for 
maintenance  trainers,  are  they  equally  valid 
for  operator  and  crew  trainers?  Do  these  indi- 
cators apply  equally  well  to  part  task  training 
devices  and  to  system  devices?  These  kinds 
of  questions  must  be  answered  overtime  before 
the  model  can  be  thought  of  as  a univeral  tool 
to  be  used  by  management. 

During  the  Phase  III  validation,  each  ser- 
vice will  be  asked  to  apply  the  model  throughout 
development  and  acquisition  of  an  emerging  sys- 
tem. Use  of  the  model  during  this  phase  will 
be  controlled  only  to  the  extent  that  each  ele- 
ment of  the  model  must  be  investigated.  (Con- 
ditions may  exist  where  one  or  more  of  the 
elements  may  not  apply  to  the  chosen  system. 
It  is  important,  however,  that  as  much  infor- 
mation as  possible  be  collected  concerning  all 
parameters.) 

The  prediction  of  performance  data  will 
clearly  be  the  most  difficult  to  validate.  This 
is  compounded  by  the  fact  that  not  enough  atten- 
tion has  previously  been  given  to  the  systema- 
tic gathering  of  such  information.  As  a result, 
there  is  a lack  of  a sufficient  data  base.  Job 
performance  data  for  the  two  systems  investig- 
ated in  this  study  were  limited  to  a compari- 
son of  WUC  items,  time  to  repair,  RETOK 
rates,  etc.  Other  performance  measures  were 
subjective.  Since  the  purpose  of  collecting 
these  data  was  to  establish  their  usefulness  as 
indicators  of  effectiveness  rather  than  to  eva- 
luate trainers,  their  use  for  this  phase  was 
acceptable.  In  the  validation  phase,  however, 
it  is  reasonable  to  estimate  probable  effective- 
ness gains  on  emerging  systems  from  using 


performance  data  from  existing  systems.  These 
estimates  could  then  serve  as  a basis  for  selec- 
ting design  features  and  justifying  design 
decisions. 

DISCUSSION 

The  model  presented  above  was  prepared 
as  an  archetype,  showing  examples  of  the  types 
of  considerations  which  need  to  be  made  in  the 
creation  of  training  devices.  Such  considera- 
tions will  aid  in  making  knowledgeable  decisions 
relative  to  cost  and  training  effectiveness  dur- 
ing device  development  and  design.  Although  the 
model  is  empirically  based  and  analytically  de- 
veloped, it  is  not  without  shortcomings  which 
primarily  reflect  the  state  of  maintenance  train- 
ing the  Armed  Services  today. 

The  concensus  of  field  personnel  is  that 
training  courses  such  as  those  being  studied 
in  this  project  are  of  insignificant  value  to  train- 
ing except  for  use  as  a general  overview.  Rather, 
many  people  feel  that  the  most  valuable  training 
is  done  during  on-the-job  training  (OJT).  This 
assumption  is  partially  supported  by  the  pre- 
vious literature  in  which  no  differences  are 
found  between  the  performance  of  students  hav- 
ing been  through  courses  using  actual  equipment 
and  those  who  went  through  courses  using  a train- 
ing device.  As  stated  in  the  introduction  to  this 
paper,  one  of  the  shortcomings  of  this  research 
is  that  the  focus  has  been  primarily  on  effective- 
ness vis  a vis  the  school  situation  rather  than 
the  actual  field  situation.  However,  the  very 
structure  of  maintenance  training  in  the  Ser- 
vices supports  this  assumption:  students  are 
taught  basic  information  in  the  school  and  more 
specific  information  through  either  formal  or 
informal  OJT  upon  graduation.  Therefore, 
all  technicians  will  eventually  reach  a minimum 
standard  of  qualification  for  the  job:  if  not,  they 
will  be  moved  into  a different  job  classification. 
As  a result,  the  question  is  not  how  effective 
is  the  training,  but  rather  how  quickly  does 
the  student  reach  the  level  of  minumum  qualifi- 
cation ( Figure  4). 


figure  4.  CompvKiv*  Hnti  wim  much  various  1 raining  Counts  Prepare 
Students  to  Matt  Minimum  Standard  of  Performance 

Such  criterion-referenced  measures  of 
training  effectiveness  may  no  longer  be  appro- 
priate since,  the  advent  of  training  devices.  It 
is  not  safe  to  assume  that  because  a training 
course  has  been  taught  in  a classroom  using 
actual  equipment  as  an  adjunct,  that  a training 
device  must  be  designed  to  fit  this  mode.  This, 
coupled  with  the  fact  that  there  have  been  vast 
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advances  in  instructional  technology,  suggest 
that  now  is  a good  time  to  reevaluate  cur- 
rent training  methods,  and  devise  a system  which 
will  enhance  effectiveness  in  an  absolute,  not 
just  a relative,  sense.  It  is  not  necessarily 
logical  to  take  the  same  course  documentation, 
as  is  currently  done,  and  exchange  the  terms 
"training  device"  for  "actual  equipment." 

There  is  another  difficulty  which  hinders 
the  development  of  devices  which  are  more 
training  effective  than  the  actual  equipment. 
Although  there  have  unquestionably  been  ad- 
vances in  the  area  of  instructional  technology 
over  the  past  ten  years,  in  the  main  there  is 
no  empirical  evidence  to  support  their  effec- 
tiveness. Researchers  have  been  pleading  for 
more  background  investigation  into  these  areas. 
However,  although  virtually  every  technical  pa- 
per on  the  topic  seems  to  end  with  a statement 
of  this  need,  the  research  itself  is  being  done 
slowly.  If  we  truly  wish  to  improve  the  quality 
of  maintenance  technicians  --  and  concomitant- 
ly the  operational  readiness  of  the  Armed  Ser- 
vices --  it  is  well  past  the  time  to  earnestly 
investigate  these  matters.  More  comprehensive 
performance  data  must  be  collected,  and  a long- 
itudinal data  base  developed  so  that  meaningful 
judgments  of  effectiveness  can  be  made. 

Some  of  the  reasons  for  the  lack  of  empir- 
ical evidence  supporting  the  worth  of  training 
devices  vis  a vis  effectiveness  have  been  dis- 
cussed in  this  paper.  However,  the  lack  of 
such  evidence  is  not  surprising:  course  approach 
does  not  change  with  the  addition  of  a trainer: 
better  instructional  technology  is  not  implement- 
ed in  trainer  design;  the  usefulness  of  the  vari- 
ous instructional  technologies  is  not  known  and 
is  not  being  researched.  Training  devices  cur- 
rently represent  an  engineering  solution  to  a 
nonengineering  problem.  This  must  be 
changed.  Creative  designs  and  solutions  in 
training  terms  must  be  applied  in  the  develop- 
ment of  such  devices,  with  the  application  of 
new  techniques  to  this  new  technology. 
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